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Abstract 

At one of Ontario’s largest universities, the University of Ottawa, course evalua¬ 
tions involve about 6,000 course sections and over 43,000 students every year. 
This paper-based format requires over 1,000,000 sheets of paper, 20,000 en¬ 
velopes, and the support of dozens of administrative staff members. To exam¬ 
ine the impact of a shift to an online system for the evaluation of courses, the 
following study sought to compare participation rates and evaluation scores of 
an online and paper-based course evaluation system. Results from a pilot group 
of 10,417 students registered in 318 courses suggest an average decrease in par¬ 
ticipation rate of 12-15% when using an online system. No significant differ¬ 
ences in evaluation scores were observed. Instructors and students alike shared 
positive reviews about the online system; however, they suggested that an in- 
class period be maintained for the electronic completion of course evaluations. 

Resume 

A l’Universite d’Ottawa, une des plus grandes universites de l’Ontario, les 
evaluations de cours impliquent quelques 6 000 cours et plus de 43 000 
etudiants chaque annee. Plus d’un million de feuilles de papier, 20 000 
enveloppes et le soutien de quelques douzaines d’employes sont necessaires 
a la tache. Pour etudier les repercussions de la transition vers un systeme 
devaluation en ligne, l’etude presentee visait a comparer les taux de participation et 
les resultats devaluation d’un systeme en ligne et d’un systeme manuel (papier). 
Les resultats d’un groupe pilote compose de 10 417 etudiants, inscrits a 318 
cours, suggerent une diminution des taux de participation moyens de 12 a 15 % 
avec l’utilisation du systeme en ligne, mais aucune difference significative dans 
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les resultats devaluation n’a ete observee. Par ailleurs, professeurs et etudiants 
ont emis des commentaires positifs sur le systeme en ligne, mais ont suggere 
de maintenir une courte periode en classe pour que les etudiants remplissent 
devaluation de leur cours en ligne. 

Course evaluations and the promotion of this practice are important parts of any sys¬ 
tem that aims to enhance teaching and contribute to the quality of the student experience. 
Not to be overlooked, feedback from course evaluations is an important tool for profes¬ 
sors. Beyond the feedback they receive, the results are also important for the development 
of their careers since they are part of a professor’s portfolio required for tenure, promo¬ 
tion, and contract renewals. 

This paper presents the results of a pilot project conducted at the University of Ot¬ 
tawa to evaluate the impact of the online evaluation of courses on student participation 
rates and on the evaluation scores when compared with the paper-based course evalua¬ 
tion method completed in class. 

Context and Review of the Literature 

At the University of Ottawa course evaluations involve about 6,000 course sections and 
over 42,000 students every year. Paper-based course evaluations consisting of 13 ques¬ 
tions are distributed in-class and completed by the attending students. Near the end of 
the semester, a 20-minute period is set aside for the completion of evaluation forms at the 
beginning of the class. Students are also given an additional sheet to share their comments 
with the course instructor. The forms are scanned by an optical reader, and the results are 
sent to the professor, the program director, and the dean. Comment sheets are only sent 
to the professor. Every year course evaluations require over 1,000,000 sheets of paper 
and 20,000 envelopes. In recent years, the average participation rate for the paper-based 
evaluations has been about 64%, ranging from 60% to 81% depending on the faculty. 

The University of Ottawa has sought to implement a new system to evaluate all cours¬ 
es online for several years, however, before doing so the administration tested the online 
evaluation format within the institutional context. In North America, interest in the on¬ 
line evaluation of courses increased significantly in the early 2000s. The percentage of 
American universities using online evaluations rose from 2 % to 33% between 2000 and 
2005 (Anderson, Brown, & Spaeth, 2006). In recent years, several Canadian universities 
have adopted, either entirely or partially, the online evaluation of courses. Many other 
universities, much like the University of Ottawa, are discussing the possibility of imple¬ 
menting such a system. 

Universities offering the online evaluation of courses make their evaluation forms 
available throughout the evaluation period (about two weeks near the end of each semes¬ 
ter). This provides students with more time to complete the evaluation and to do so at a 
time convenient to them. Some institutions have maintained a mandatory in-class period 
to allow students to complete the evaluation in class with their laptop or mobile devices, 
while other universities have not retained the in-class system. 

Research into the online evaluation of courses has attempted to address the university 
community’s main concerns in this area. These include participation rates, course evalua¬ 
tion scores, comments provided by students, and incentives to fill out the evaluations. The 
following section highlights literature related to each of these areas of concern. 


CJHE / RCES Volume 47, No. 2, 2017 



The Online Evaluation Of Courses / J. F. Groen & Y. Herry 108 


Participation Rates 

The participation rate for online evaluations is the foremost concern identified in the 
literature (Adams & Umbach, 2012). Indeed, most studies on the subject point to a lower 
participation rate for the online evaluation of courses, compared with paper-based evalu¬ 
ations (Gamliel & Davidovitz, 2005; Nevo, McClean, &Nevo, 2010; Hativa, 2013; Stowell, 
Addison, & Smith, 2012). The participation rate for online evaluations varies among insti¬ 
tutions, with rates between 30% and 53% (Nowell, Lewis, & Handley, 2010), the majority 
hovering around the 50% mark. Some universities have achieved participation rates of 
nearly 70% by offering student incentives. 

Universities that have moved from paper-based evaluations to online evaluations of¬ 
ten note a slight increase in the participation rate of online evaluations over a multi-year 
period (Nevo et al., 2010). However, research has shown that technology alone is not the 
sole influencing factor for the participation rate, rather it is the engagement of academic 
leaders that has a significant impact on the level of participation (Pitre-Hayes, 2013). The 
more students see the effects of these evaluations on the quality of teaching, the higher 
participation rates will be. Two factors reducing the rate of participation. First, the stu¬ 
dent perception that an evaluation is only useful to the professor, and second, students 
will not benefit from their course evaluation results because they will have now finished 
the course (Nevo et al., 2010). There is, therefore, continued work to do in terms of ex¬ 
plaining and demonstrating the benefits of course evaluations. 

Despite the factors listed above, it is understood that certain groups of students are 
more likely to complete course evaluations, whether online or paper-based. Generally, 
these are female students and academically strong students (based on course marks and 
cumulative GPA) (Adams & Umbach, 2012; Hativa, 2013). 

Course Evaluation Scores 

Lower participation rates in course evaluations raise the following question: how rep¬ 
resentative are the students who complete the questionnaire compared with the overall 
student body taking the course? Professors and researchers wonder if certain groups— 
such as the least satisfied students, the lowest achieving students, or students least en¬ 
gaged in the course—are more likely than others to answer the questionnaire online, 
(Hativa, 2013). A larger proportion of these students would therefore have a negative 
impact on the evaluation scores. If one subgroup of students is more likely to participate 
than another, it could be presumed that online evaluations introduce a bias at the expense 
of the latter group. However, research on this subject shows that this is not the case, and 
that these ideas represent misconceptions about the online evaluation of courses (Uni¬ 
versity of Saskatchewan, n.d.). In fact, according to several studies, the judgements made 
by students about courses through online evaluations appear to be similar to those made 
using paper-based evaluations, and this despite a lower participation rate (Gamliel & Da¬ 
vidovitz, 2005; Legg & Wilson, 2012; Nowell et al., 2010; Stowell et al., 2012; University 
of Saskatchewan, n.d.; Venette, Sellnow, & McIntyre, 2010). 

Burton, Civitano, and Steiner-Grossman (2012) conducted a meta-analysis of research 
on the subject identifying 16 studies that compared the results of online evaluations with 
those of paper-based evaluations. Fourteen of these studies found no significant difference 
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between the two methods of evaluation, and two studies found a slight increase in positive 
scores for online evaluations. Burton, et al. (2012) also found increased positive scores in 
online evaluations, as did Nowell et al. (2010) and Morrison (2013). The research results do 
not support the idea that the online format negatively influences course evaluation scores. 

Comments Provided by Students 

One of the positive effects of online evaluations is the quality and quantity of com¬ 
ments from students, regarding their educational experience (Crews & Curtis, 2011; Legg 
& Wilson, 2012). Studies have shown that lengthier comments have been associated with 
online evaluations (Venette et al., 2010). The comments were more developed, better 
constructed (Morrison, 2011), more positive, and more useful (Burton et al. 2012; Heath, 
Lawyer, & Rasmussen, 2007) than those provided in the paper-based evaluations. In ad¬ 
dition, a larger number of students were said to provide comments when using the online 
evaluation format. 


Incentives to Increase Participation Rates 

Researchers have studied cases where incentives were offered to boost course evalu¬ 
ation participation rates. Among those analyzed, the mandatory periods of class time de¬ 
voted to the completion of the course evaluations was maintained, which confirmed in 
the eyes of students that completing the course evaluation was important enough to merit 
class time (Crews & Curtis, 2011; Nevo et al., 2010). In addition, students noted that com¬ 
pleting course evaluations outside of class time represented an extra task for them. 

Other incentives are used by some universities, but implementing these involves a 
number of challenges in terms of both management and ethics. Among the more extreme 
examples, we find the mandatory evaluation of courses and imposition of penalties such 
as withholding final marks until the course evaluation has been completed (Linse, 2012). 
Other universities have used more positive methods such as adding marks (between one 
and five) to the final marks of students who have completed the course evaluation (Pitre- 
Hayes, 2013; University of Saskatchewan, n.d.). Crews and Curtis (2011) compiled a list of 
the most common strategies used by professors who obtained course evaluation partici¬ 
pation rates of over 80%. It should be noted that these professors often combined several 
strategies. Some 70% of this group had sent a personal email to their students to high¬ 
light the importance of the course evaluations, and 52% had given extra marks for course 
evaluations. In another study (University of Saskatchewan, n.d.), professors who made the 
completion of the course evaluation an assignment and gave students additional marks 
obtained an average participation rate of 87%. This percentage fell slightly to 77% for pro¬ 
fessors who gave the evaluation as an assignment but did not give marks; professors whose 
only strategy was to mention the course evaluations to their students had an average par¬ 
ticipation rate of 32%, and those who failed to mention it at all had a 20% participate rate. 

Another strategy that encourages student participation in course evaluations is the 
use of an informal midterm course evaluation. Research shows that professors who ask 
their students’ opinions on the course at midterm and who take their comments into ac¬ 
count during the rest of the course have high participation rates during the final course 
evaluation (Davis, 2009; University of Saskatchewan, n.d.; University of Ottawa, n.d.). 
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In conclusion, it must be said that all of the strategies that involve requiring evalua¬ 
tions to be completed or the allocation of additional marks are very difficult to manage; 
they can endanger the confidentiality of student responses and can introduce bias in the 
evaluation scores because students may only complete the form to get additional marks 
without giving the questions their fullest attention. However, no study to date has looked 
at the impact of these strategies on evaluation scores. 

Pilot Project Research Questions 

Based on the results of this literature review, the University of Ottawa put in place a 
pilot project to address the following questions: 

• What is the impact of an online evaluation format on participation rates and course 
evaluation scores? 

• What is the level of satisfaction of users? 

• When did students complete the course evaluations and using what type of device? 
The methodology of the survey was developed to answer these questions. 

Methodology 


Instruments 

The instruments used to collect data from students were the course evaluation form 
and a separate student questionnaire on their level of satisfaction with the online process 
and the way in which they completed the course evaluation. Data collected from profes¬ 
sors included a questionnaire regarding their level of satisfaction with the evaluation pro¬ 
cess and about the results obtained. This questionnaire was administered following the 
receipt of student course evaluation results. 

Course evaluation form. The standard course evaluation form at the University of 
Ottawa consists of 13 questions. The majority of course evaluations are carried out using a 
paper-based format that is distributed to students in class. However, the university offers 
an online version of the questionnaire to students in all its distance education courses and 
students with special needs. The online questionnaire includes the same questions as the 
paper-based version and is identically formatted. As with the printed version, it contains 
13 questions with an additional comment section. The online version of the form was used 
for the purposes of this project. 

Student questionnaire. In addition to the course evaluation form, each student 
was asked to answer six questions on how the evaluation was carried out, their level of 
satisfaction with the online evaluation process, and the device they used to complete the 
questionnaire. Students responded to a separate online questionnaire immediately after 
completing the course evaluation. The questions were as follows: 

1. I completed the online evaluation of this course (When I received the email invite; 
during the in-class period; when I received the email reminder; at another time). 

2. What device are you using to complete the online evaluation of the course? (Tablet 
[iPad, etc.]; smartphone; personal computer; campus computer). 

3. I found it easy to use the online evaluation format. (Using a four-point Likert scale). 

4.1 was able to give more detail about my experience in the course using the online 

evaluation format than the paper-based format. (Using a four-point Likert scale). 
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5. I would feel comfortable using the online evaluation format for all of my course 
evaluations. (Using a four-point Likert scale). 

6. If you could improve one thing about the online evaluation format, what would you 
suggest? (Access to the questionnaire via uoZone, on-campus Wi-Fi connection, 
navigation using a smartphone/tablet, other [please elaborate below]; no improve¬ 
ment needed). 

Lastly, students were given a section to provide additional comments. 

Questionnaire for professors. After receiving the online evaluation results (scores 
and student comments), professors who participated in the project were sent a five-ques¬ 
tion survey to be answered using a four-point Likert scale ranging from strongly agree to 
strongly disagree. The questions focused on how the online evaluation was carried out 
and their level of satisfaction. The questions were as follows: 

1. Students provided more detail in the open-ended comments using the online eval¬ 
uation format compared to the paper-based format. 

2. Student comments were of higher quality in the open-ended comments using the 
online evaluation format compared to the paper-based format. 

3. Student comments were more useful to my teaching in the open-ended comments 
using the online evaluation format compared to the paper-based format. 

4.1 would feel comfortable using the online evaluation format for all of my 
course evaluations. 

5. Should the in-class evaluation completion period be maintained? (Yes/No). 

Lastly, professors were given a section to provide their comments. 

Participants 

The aim of the pilot project was to target a representative sample of the courses of¬ 
fered by the university. To compile this sample, the registrar’s office provided a list of 400 
courses for the fall 2013 session and a list of 325 courses for the winter 2014 session. The 
sampling took into consideration the course year (1st year through 4th year and gradu¬ 
ate studies) and the number of students registered in each course. Because participation 
in the project was voluntary, the professor of each of the courses in the sample was sent 
an email asking them to participate in an online evaluation of their course rather than 
using the paper-based method. Of the 400 fall session courses, 180 professors agreed to 
take part in the project; of the 325 winter session courses, 138 professors accepted. In 
total, 10,417 students participated in the project and 4,508 completed the six-item ques¬ 
tionnaire regarding their perception of how the online evaluation was carried out. Every 
faculty (except education, which has a different course evaluation period) and grade level 
was represented in the study’s respondents. During the data analysis, the averages were 
adjusted to take into account the new distribution of classes, according to their represen¬ 
tativeness of all of the courses offered over the two sessions. 

Data Collection 

Data collection was carried out between November 2013 and April 2014. The evalu¬ 
ation form was made accessible online using a secure website throughout the regular 
course evaluation period, two weeks in November for the fall session, and two weeks in 
March for the winter session. During this period, students were able to complete the form 
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at a time convenient to them. The mandatory in-class period was also maintained and 
students who had not yet completed the course evaluation had the choice to use this in- 
class time to complete the course evaluations using a personal computer, smartphone, or 
tablet while the course instructor stepped out of the room. After answering the 13 ques¬ 
tions on the course evaluation form and saving their responses, students could then an¬ 
swer the six additional questions regarding how the evaluation was carried out. At the 
beginning of the regular course evaluation period, the system sent an email to students 
inviting them to complete the evaluation for their course, followed by an email reminder 
and a thank-you message at the end of the evaluation period. Student responses were 
confidential. Anonymous student comments were made available to professors via their 
personal faculty profile accounts. 

Data Analysis Plan 

The analysis of results can be divided into three main sections. The first is the com¬ 
parison of the results obtained by professors in the online evaluation project with the 
results obtained using the paper-based evaluations from the same course taught by them 
at least once over the last three years. During this stage of the data analysis, 182 courses 
participating in the pilot project were identified as having carried out a course evalua¬ 
tion using the paper-based format in the three years preceding the project. Each of these 
courses were taught by the same professor, were subject to the online evaluation as part 
of this project, and had at least one paper-based evaluation in the past three years. The 
participation rate and the course evaluation scores obtained during the online evaluation 
of the 182 courses were compared to those obtained by the same courses evaluated using 
the paper-based method. This comparison is the most statistically robust since it controls 
for a large number of associated variables including the course code; the course level; the 
faculty and department offering the course; the professor teaching the course; the profes¬ 
sor’s gender, status, and academic ranking; and the students registered for the course. 

The second major section of the data analysis compared the results obtained by all 
courses in the 2013-2014 online project with the results from courses that carried out the 
paper-based evaluation in the same academic year. During this stage of the data analysis, 
the participation rates and the evaluation scores of 318 courses evaluated online were 
compared with 3,516 courses that were evaluated using the paper-based method during 
the same period. This comparison is statistically less robust than the previous one since it 
does not control for as many associated variables but does allow control of the period of 
evaluation, something that the previous analyses did not permit. 

A common statistical method used to test differences between two or more groups is 
the analysis of variance (ANOVA). However, when the hypothesis of independence and 
homogeneity are not fulfilled, neither ANOVA nor Kruskal-Wallis test (a non-parametric 
version of ANOVA) apply due to the high risk of false positive or false negative results. 
Given the absence of normality, homogeneity, and the non-independent nature of the 
sample in this study, the Wilcoxon signed-rank test was used to compare the difference 
between the means of two groups. 

The third major section of the data analysis focused on the presentation of frequency 
distributions of responses provided by students and professors in their respective question¬ 
naires. This helped review trends that informed the formulation of several recommendations. 
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Results 

The presentation of the results is divided into four main sections. The first outlines a 
comparison between the participation rates of the online evaluation and the paper-based 
evaluation. The second outlines a comparison between the evaluation scores obtained by 
professors from the online and paper-based evaluations. The third and fourth sections 
outline the frequency distributions of responses provided by students and professors in 
their respective questionnaires. 

Comparison of Online and Paper-Based Evaluation Participation Rates 

Table 1 outlines the participation rates achieved by all courses, based on the level of 
study (undergraduate or graduate) and the year of study (first, second, third, or fourth 
year). The results reveal statistically significantly lower participation rates for online eval¬ 
uations than for paper-based evaluations [S = 5,308, p < .0001]. In the comparison of 
evaluations for the 182 courses (comparison 1), the online participation rate was 51% and 
63% for the paper-based evaluations. In the comparison between the 318 courses partici¬ 
pating in the pilot project and all other courses (total of 3,516) evaluated during that same 
period (comparison 2), participation rates were 51% and 66% respectively. 

Table 1. Comparison of the Participation Rate by Level and Year of Study 




Comparison 1 

Comparison 2 

Year of Study 

Online 

course 

evaluations 

(182) 

Most recent 
paper-based 
course evalua¬ 
tions (182) 

Online 

course 

evaluations 

(318) 

Total courses 
evaluated in 
2013-2014 

Total 

Total number of 

courses 

182 

182 

318 

3,516 


Total number of 
students 

5,842 

7,370 

10,417 

174,863 


% students evaluated 

51% 

63% 

51% 

66% 


(standard deviation) 

(0.20) 

(0.16) 

(0.20) 

(0.27) 

1 st Year 

Number of courses 

35 

35 

56 

312 


Number of students 

1,729 

2,309 

2,935 

55,651 


% of respondents 

51% 

58% 

48% 

61% 


(standard deviation) 

(0.14) 

(0.16) 

(0.16) 

(0.18) 

2 nd Year 

Number of courses 

49 

49 

77 

691 


Number of students 

1,576 

2,040 

3,032 

47,921 


% of respondents 

49 % 

65% 

48% 

65% 


(standard deviation) 

(0.19) 

(0.17) 

(0.18) 

(0.43) 
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Comparison 1 

Comparison 2 

Year of Study 

Online 

course 

evaluations 

(182) 

Most recent 
paper-based 
course evalua¬ 
tions (182) 

Online 

course 

evaluations 

(318) 

Total courses 
evaluated in 
2013-2014 

3 rd Year 

Number of courses 

58 

58 

98 

963 


Number of students 

1,894 

2,144 

2,981 

39,780 


% of respondents 

53 % 

62% 

53 % 

69% 


(standard deviation) 

(0.18) 

(0.16) 

(0.18) 

(0.18) 

4 th Year 

Number of courses 

24 

24 

55 

705 


Number of students 

471 

685 

1,060 

17,872 


% of respondents 

50% 

75 % 

56 % 

69% 


(standard deviation) 

(0.23) 

(0.07) 

(0.22) 

(0.18) 

Gradu¬ 

ate 

Total number of 

courses 

16 

16 

32 

845 

Studies 

Total number of 
students 

172 

192 

409 

13,639 


% students evaluated 

72% 

84% 

72% 

80% 


(standard deviation) 

(0.22) 

(0.12) 

(0.19) 

(0.19) 


The comparison between levels of study allows us to observe that a similar difference 
of 12-14% between the online and paper-based evaluation participation rates is pres¬ 
ent across undergraduate levels and for graduate studies. The participation rate is higher 
by 20% for graduate courses than for undergraduate courses, regardless of whether the 
evaluations were conducted online or in print. Online evaluation participation rates are 
relatively constant depending on the year of study (first, second, third, or fourth) varying 
from 48-56%, whereas paper-based evaluation participation rates fluctuate more based 
on the year of study, varying from 58-75%. An analysis of participation rates for online 
evaluations in each faculty are lower, varying from 43-82%, whereas participation rates 
for paper-based evaluation vary from 53-84%. 

Comparison of Online and Paper-Based Evaluation Scores 

Table 2 outlines the results from both comparison groups based on the level of study 
(undergraduate or graduate), year of study (first, second, third, or fourth). The results 
reveal similar evaluation scores for online evaluations (average of 4.0/5) and for paper- 
based evaluations (average of 3.9/5). When examining scores based on the level of study, 
year of study or by faculty, no significant difference is revealed between the scores ob¬ 
tained using the online format and the paper-based format [S = -933.5, p = 0.2014]. 
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Table 2. Comparison of the Course Evaluation Scores by Level and Year of Study 




Comparison 1 

Comparison 2 

Year of Study 

Online 

course 

evaluations 

(182) 

Most recent 
paper-based 
course evalua¬ 
tions (182) 

Total online 
course sample 

(318) 

Total courses 
: evaluated in 
2013-2014 

Total 

Total number of courses 

182 

182 

318 

3,516 


Total number of stu- 

5,842 

7,370 

10,417 

174,863 


dents 

Mean score 

4.0 

3-9 

4.0 

4.0 


(standard deviation) 

(0.60) 

(o. 57 ) 

(0.61) 

(o. 59 ) 

1 st Year 

Number of courses 

35 

35 

56 

312 


Number of students 

1,729 

2,309 

2,935 

55,651 


Mean score 

3-9 

3-9 

4.0 

4.0 


(standard deviation) 

(0.56) 

(o. 55 ) 

(0.56) 

(0.58) 

2 nd Year 

Number of courses 

49 

49 

77 

691 


Number of students 

1,576 

2,040 

3,032 

47,921 


Mean score 

3-9 

3-9 

3-9 

4.0 


(standard deviation) 

(o. 57 ) 

(o. 57 ) 

(o. 59 ) 

(0.58) 

3 rd Year 

Number of courses 

58 

58 

98 

963 


Number of students 

1,894 

2,144 

2,981 

39,780 


Mean score 

4-1 

3-9 

4.0 

4.0 


(standard deviation) 

(0.66) 

(0.61) 

(0.64) 

(o. 59 ) 

4 th Year 

Number of courses 

24 

24 

55 

705 


Number of students 

471 

658 

1,060 

17,872 


Mean score 

4.2 

4.2 

4.0 

4-1 


(standard deviation) 

(o. 53 ) 

(0.45) 

(0.61) 

(o. 57 ) 

Graduate 

Total number of courses 

16 

16 

32 

845 

Studies 

Total number of stu¬ 

172 

192 

409 

13,639 


dents 

Mean score 

4-3 

4-4 

4.2 

4.2 


(standard deviation) 

(0.61) 

(0.48) 

(0.62) 

(0.60) 
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Student Questionnaire Responses Regarding Level of Satisfaction and the 
Evaluation Process 

We learned from students’ questionnaire responses that 97% of them found online 
evaluations easy and that 91% of them felt comfortable evaluating all of their courses 
online. As for the written comments, 65% found that online evaluations allowed them to 
write more detailed comments. 

For the time given to complete the questionnaire, almost half of the students (47%) 
completed the course evaluation within the dedicated in-class period. Almost a quarter of 
those students (23%) completed the questionnaire online when they received the initial 
email inviting them to do so, and 8 % completed it when they received the reminder email. 
The rest of the students completed the questionnaire at different times during the two-week 
course evaluation period. Eighty percent of students used their personal computers to com¬ 
plete the questionnaire, while the rest used a tablet, smart phone, or university computer. 

Regarding possible improvements to the evaluation system, 60% of students had no 
recommendations. However, 10% of students asked for an improvement with the use of 
smartphones. This percentage represents the number of students who used this tool to 
complete the questionnaire. Lastly, 13% of students asked for an improved Wi-Fi con¬ 
nection at the university, and another 13% asked for easier access to the questionnaire 
through the student portal. 

Professor Questionnaire Responses Regarding Level of Satisfaction and the 
Evaluation Process 

Professors’ questionnaire responses showed that 80% are open to or are in favour of 
the online evaluation format. Regarding the comments received on their teaching, 37% 
found that they were more detailed, 33% found them of better quality, and 29% found 
them more useful than comments provided with the paper-based evaluation. Depending 
on the questions, a proportion, from 34% to 50% of professors, did not see a difference. 

On the question of maintaining the in-class period to complete the online evaluation, 
62% of professors from the fall session were in favour of keeping it, compared to only 15% 
from the winter session. This was the only difference found in the professors’ responses 
between the two sessions. 

Summary, Discussion, and Recommendations 

The goal of this project was to evaluate the potential impact of an online format for 
the evaluation of courses on student participation rates and to compare online and paper- 
based course evaluation scores. This project aimed to answer the following questions: 

• What is the impact of the online evaluation of courses on the participation rates and 
scores of the evaluations? 

• What is the level of satisfaction of users? 

• When did students complete the evaluation, and what type of device did they use? 

The project included unique elements such as a large sample of 318 courses, each of¬ 
fered by a different professor; representation across each faculty and nearly all programs; 
and representation of all the course offerings of the university during the fall 2013 and 
winter 2014 sessions. The large number of participating students, more than 10,400, rep- 
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resenting a quarter of the student population of the university, was also different from 
previous studies. Much of the research analyzed in the literature review had smaller sam¬ 
ples and often focused on a limited number of courses that were frequently offered in 
one particular program or faculty. Another unique element of the study was the compari¬ 
son between 182 courses evaluated online and the paper-based evaluations for the same 
course taught by the same instructor within the past three years. 

The pilot project results revealed significantly lower participation rates for online eval¬ 
uations than for paper-based evaluations. These results are consistent with the results of 
other studies on the subject (Adams & Umbach, 2012; Gamliel & Davidovitz, 2005; Nevo 
et al., 2010; Hativa, 2013; Stowell et al., 2012) that have revealed lower participation rates 
for online evaluation of courses. However, the rates observed in this project can be plot¬ 
ted in the upper part of the spectrum of those studies where participation rates vary from 
30-53% (Nowell et ah, 2010). 

It should be noted that despite lower participation rates, the results revealed similar 
teaching evaluation scores for online evaluations (average 4.0/5) and paper-based evalu¬ 
ations (average 3.9/5). When examining the scores based on the level of study, year of 
study, or by faculty, no significant differences were found between the scores of online or 
paper-based course evaluations. These results are also consistent with findings from pre¬ 
vious studies, indicating that students’ experiences with online and paper-based course 
evaluations are similar, despite a decrease in the participation rates (Burton et ah, 2012; 
Gamliel & Davidovitz, 2005; Legg & Wilson, 2012; Nowell et ah, 2010; University of Sas¬ 
katchewan, n.d.; Stowell et ah, 2012; Venette et ah, 2010). 

One of the positive effects of the online evaluation of courses identified in the litera¬ 
ture review is the quality and quantity of comments provided by students on teaching 
(Crews & Curtis, 2011; Legg & Wilson, 2012). Using the online evaluation format, the 
quantity of student comments increased (Stowell et ah, 2012; Venette et ah, 2010). They 
were also more detailed, better constructed (Morrison, 2011), more positive, and deemed 
more useful (Burton et ah, 2012; Heath et ah, 2007) than comments from the paper- 
based evaluations. In addition, a larger number of students provided comments using the 
online evaluation. The results of the pilot project align closely with these listed benefits, 
as seen in the feedback provided in the survey administered to professors. 

Keeping the in-class period to complete the online evaluation did not receive strong 
support from professors. In the comments provided by professors, we identified two main 
reasons for this result. First, professors felt that because some students had already com¬ 
pleted the questionnaire, they showed up for class late, which was disruptive, and second, 
some professors felt that the time reserved for completing course evaluations would be 
better used for teaching and learning. However, we have to keep in mind that the research 
clearly indicates that the in-class period is a strong incentive for students to complete the 
questionnaire, and that in the pilot project, close to half of the students (47%) completed 
the questionnaire during this period. Students saw this time as an incentive because it con¬ 
firms that course evaluations are important enough to put time aside in class to do them, 
and they view having to do course evaluations outside of class time as an extra chore. Many 
studies recommend keeping the compulsory in-class period for online evaluation of cours¬ 
es. This recommendation was included in the report to the University of Ottawa senate. 
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The encouraging results from this pilot project raise several questions to which we 
do not have complete answers. For example, many of the universities that have adopted 
the online evaluation of courses have noted a slight increase in participation rates over 
the years (Nevo et al., 2010). Would this increase also occur at the University of Ottawa? 
Moreover, for the purpose of this project, the vast majority of the students only had one 
course to evaluate online; the other courses were evaluated in class using the paper-based 
method. Would having to evaluate five courses instead of just one have an impact on the 
participation rate and on the quantity and quality of the comments provided? On this last 
point, several studies have found that the online evaluation method has a positive impact 
on student comments (Crews & Curtis, 2011; Legg & Wilson, 2012). These are important 
questions. We would remind readers that although the participation rate in the current 
pilot project was lower for online evaluations than for paper-based evaluations, course 
evaluation scores were similar between evaluation methods, and a positive effect was ob¬ 
served on comments provided to professors. 

Recommendations made and passed by the university senate were informed by the 
findings of this study. The senate recommended that: 

• the evaluation of the course remains voluntary for students and that no incentive 
related to the allocation of additional marks should be put in place; 

• the results of courses evaluation be more readily available on the University of Ot¬ 
tawa website; 

• a significant marketing campaign for online evaluation of courses be put in place 
to encourage students to complete the evaluation forms, explain and demonstrate 
the benefits of courses evaluation, and emphasize the fact that course evaluations 
benefit students as much as the faculty and the institution; 

• professors adopt voluntary strategies to promote student participation in online 
evaluation of courses, such as the use of an informal midterm evaluation, sending 
a personal email on behalf of the professor to students to make them aware of the 
importance of course evaluations, assign the completion of a course evaluation as 
homework without the provision of extra points, integrate a course evaluation in 
the course outline, and demonstrate the changes that were implemented based on 
the feedback received; and 

• the compulsory in-class evaluation period be maintained for two years after the 
implementation of the online system and an assessment be conducted at the end of 
this period.’#' 
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